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ABSTRACT 


The purpose of this thesis is to analyze the application of Modeling and Simulation 
(M&S) within the Army Operational Test and Evaluation (OT&E) process in support of 
weapon systems acquisition. This thesis considers the Army’s current acquisition process, 
M&S technologies, infrastructure, and policies that guide the Program Manager (PM) in the 
application of modeling and simulation in operational testing. An analysis of the potential 
strengths and weaknesses of M&S in addressing OT&E issues is presented. Lessons learned 
from past OT&E efforts are analyzed for process improvement through M&S applications. 
The analysis indicates that M&S is a viable tool for assisting the PM in completing OT&E. 
M&S techniques can assist in test design validation, expand testing in areas of limited 
resources and environmental concerns, and validate live testing data. From this analysis, a 
set of recommendations are formulated, indicating where the PM can integrate M&S into the 


OT&E process. 
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I. INTRODUCTION 


A. PURPOSE 

The purpose of this thesis is to analyze the application of Modeling and Simulation 
(M&S) within the Army Operational Test and Evaluation (OT&E) process in support of weapon 
systems acquisition. The focus is to provide Program Managers (PMs) a viable tool in managing 
the Army OT&E process. From this analysis, a set of recommendations in the area of M&S 
integration into Army OT&E are formulated and offered. The recommendations address both the 
cost effectiveness and the adequacy of the application of M&S within the Army OTE process. 
B. BACKGROUND 

The modern battlefield requires continuous advancement of technology in virtually all 
complex weapon systems. These complex systems provide accuracy, reduce manpower 
requirements, and create the ability to control vast numbers of tasks simultaneously. However, 
because of their complexity, they pose a challenge to the OT&E process due to the high cost and 
difficulty of testing them in an operational environment. These difficulties arise from the 
complex nature of weapons on the modern battlefield and the need to maintain “technologically 
current” weapon systems. Two examples include weapons tracking systems capable of 
identifying and tracking multiple targets and communications systems capable of simultaneously 
processing thousands of analog and digital messages. 

Within the Army OT&E process of weapon systems development, M&S is an emerging 
technology that is available for application in the defense acquisition process. Specifically, 


within the Army OT&E process, this technology offers great promise to decrease the amount of 


time and money spent. As the defense budget continues to rapidly decline, technologies 
comprising revolutionary techniques in M&S are receiving increasingly wider interest and 
application within the Army OT&E process. The use of M&S in the Army OTE process of 
weapon systems acquisition can help the PM analyze and identify potential areas of concern 
before significant funds are spent in live testing and advanced development. Early use of M&S 
in the Army OT&E process can provide unique insight into system problems. As the cost of 
OT&E continues to increase with the complexity of today’s weapon systems, the application of 
M&S in the Army OT&E process becomes ever more appealing. 

minteialencine the appeal of the relatively low cost of M&S for OT&E applications 
are laws, directives, regulations, and policies mandating live testing. In the test and evaluation 
community, there is a widely held distrust for anything that 1s “simulated,” including simulation 
used in OT&E. The validity of the simulation model is a primary concern in developing viable 
information and reliable test data for the PM. A critical issue that must be considered in 
developing the use of M&S in the Army OTE process 1s the suitability of the simulation model 
in replicating the real-world environment. Simulation applications have the potential to evaluate 
virtually every component of a weapon system, in all environments, providing large amounts of 
data to assist the PM in managing weapons system acquisition. Accredited M&S techniques, 
used in the Army OT&E process, have the potential to assist the PM in delivering combat-ready 
weapon systems at significant reductions in both cost and time to delivery. 
C. THESIS OBJECTIVE 

This thesis analyzes the application of M&S in the Army OT&E process in support of 


weapon systems acquisition. The current use of M&S in the Army’s OT&E process is analyzed 


to identify persistent problems and issues, and assess shortcomings in its application. Analysis is 
offered where M&S concepts have potential application for enhancing the Army OT&E process 
as to the type, extent, and applicability of M&S concepts. 

Analysis of the potential strengths and weaknesses of modeling and simulation in 
addressing the elements and issues associated with the Army OT&E process are presented. 
Lessons learned from past OT&E efforts are analyzed for possible process improvement through 
M&S integration. Supporting conclusions and recommendations based on the analysis of the 
thesis are presented. Areas for further research are identified to focus future study. 

D. RESEARCH QUESTIONS 
The primary research question for this thesis 1s: 


1. What current role does M&S play and what future role should it play in support 
of the Army OT&E process? 


The subsidiary research questions are: 
1. How does M&S support the Army OT&E process? 


2. What regulatory and policy guidelines are established for governing the use of M&S 
in the Army OT&E process? 


3. What types of M&S are currently being used within the Army OT&E process? 
4. What types of Army OT&E methods lend themselves to the use of M&S? 


5. What are the key elements to consider when using M&S in the Army OT&E 
process? 


6. What are the capabilities of M&S in relationship to the Army OT&E process? 


7. What type of information can be gained from using M&S? 
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E. SCOPE AND LIMITATIONS 

The thrust of this thesis concerns M&S applications within the Army OT&E process in 
support of weapon systems acquisition. The potential issues associated with the application of 
M&S in this thesis are limited to the OT&E process. Specifically, this thesis addresses the 
OT&E process rather than the Developmental Test and Evaluation (DT&E) process. 

This thesis identifies potential opportunities to integrate M&S in the Army OT&E 
process. The development of a common foundation of knowledge that governs the Army OT&E 
process will establish the various applications of M&S as an integral part of OT&E. The laws, 
regulations, directives, and policies pertaining to Army OT&E are examined to identify where 
and under what circumstances M&S integration can be accomplished and where it 1s prohibited. 
Next, a basic understanding of M&S techniques and applications will establish the plausibility of 
using M&S in the Army OT&E process. Finally, current Army acquisition programs are 
analyzed to evaluate specific successes and failures when integrating M&S into the Army OT&E 
process. 

F. RESEARCH LITERATURE AND METHODOLOGY 

Research data were obtained from official Government directives and policies, journals, 
previous theses, United States Code, DoD and Army regulations and manuals, and personal 
interviews. Information on current M&S programs was obtained from Army Training and 
Doctrine Command (TRADOC), Analysis Centers (TRACs), the Army’s Simulation, Training 
and Instrumentation Command (STRICOM), and the Defense Modeling and Simulation Office 
(DMSO). Information on the U.S. Army Operational Test and Evaluation process was obtained 


from the Operational Test and Evaluation Command (OPTEC) and the Test and Experimentation 


Command (TEXCOM). Current directives and policies guiding the Army OT&E process were 
reviewed. A comprehensive literature search was conducted to determine the existence and 
validity of available simulation models and techniques. 

Research was conducted via personal and telephone interviews with cognizant M&S and 
Test and Evaluation (T&E) personnel. Interviews with OT&E related individuals concentrated 
on the opportunities for M&S to enhance the OT&E process. Interviews with M&S-related 
individuals focused on the state-of-the-art in M&S and its applicability to the Army OT&E 
process. 

G. ACRONYMS 

An extensive listing of acronyms associated with both OT&E and M&S subjects is 
presented in the Appendix. 

H. ORGANIZATION OF THESIS 

Chapter I contains the purpose, background, thesis objective, research questions, scope 
and limitations, methodology, and thesis organization. 

Chapter II of this thesis addresses the acquisition process and how the Army OT&E 
process supports new weapon systems acquisition. It explores the relationship between the Army 
DT&E process and OT&E process, establishing the interface between M&S and the Army 
OT&E process. 

Chapter III explores the basics of M&S as a tool for assisting the PM. It establishes a 
baseline of information about M&S including key elements in simulation, types of simulation 
available, capabilities of simulation, and types of information which simulation provides the 


PM. 


Chapter IV outlines the Army’s current guidelines and policies that affect the use of M&S 
in the Army OT&E process. It then compares and contrasts Army directives with those 
established by DoD. Current facilities that support M&S are investigated to assist the PM in 
using simulation in the Army OT&E process. 

Chapters V and VI contain analysis of M&S applications in actual weapon systems and 
provides recommendations for future applications of M&S inthe Army OT&E process. Areas 


for further research are also identified. 


Il. THE ACQUISITION PROCESS AND OT&E 


A. GENERAL 

The United States and the Army are facing a dynamic international environment which 
involves a complex array of security issues, unpredictable threats and a fiscally restrained 
domestic budget. In this environment, the Army needs to draw on its defense acquisition 
professionals to help overcome formidable challenges of rapidly fielding technically superior 
battlefield weapon systems. In an address to the National Contract Management Association in 
1993, Dr. William J. Perry, as Deputy Secretary of Defense, explained the importance of fielding 
technologically superior weapon systems. He stated, “The advantage we had in Desert Storm 
had three components. We had an advantage in people, an advantage in readiness, and an 
advantage in eennoleo (Piplani, 1994, p. v) The technological advantage achieved by the 
military is maintained through the acquisition process. The acquisition process is the primary 
means through which the Army procures new equipment and technology to maintain the 
“technology advantage.” 

Operational Test and Evaluation (OT&E) is a critical component of the acquisition 
process that ensures weapon systems are fielded in a combat-ready status. As part of the 
acquisition process, OT&E is designed to assess a system's operational effectiveness and 
suitability in a realistic operational environment. The most capable system is useless if it cannot 
be employed by the intended user within doctrinal, tactical, force structure, and training 


requirements. 


Modeling and Simulation (M&S) offers a new technology that will streamline the time 
and funding spentin OT&E. Simulation and modeling techniques can be applied to every major 
Army weapon development program to reduce design and production cost, improve performance, 
improve diagnostics and maintenance, assist in better and faster training of personnel, and 
improve command and control on the battlefield. (Piplani, 1994, p. v) 

The purpose of this chapter is to briefly review the Army acquisition process and delineate 
the role of OT&E within the acquisition process. The chapter will conclude with the integration 
of M&S in support of OT&E. 

B. THE ACQUISITION PROCESS 

The roots of the current defense systems acquisition process can be traced to the early 
1950s. Over the years many reform initiatives have been implemented to make the acquisition 
process more efficient and effective. However, the goal of the acquisition process has remained 
constant throughout the years, “... to procure state-of-the-art technology and products, rapidly, 
from reliable suppliers who utilize the latest manufacturing and management techniques that 
satisfies a specific user’s need at an affordable cost.” (Piplani, 1994, p 1-4) The policy 
foundation for the acquisition process 1s found in the DoD 5000 series. 

The DoD 5000.1 establishes broad policies governing the defense systems acquisition 
process. It states that the three decision-making support systems must interact and interface with 
each other in order for the process to work effectively. The three systems illustrated in Figure 1 
are: 1) requirements generation, 2) acquisition management and 3) planning, programming and 
budgeting (PPBS). This triad of effort is supported by a structured program of checks and 


balances. Figure 2 depicts the milestones in the acquisition process. Within the structure of the 


acquisition process, test and evaluation verifies the maturity of system design and functionality of 


system work performed during each milestone. 
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Figure 1. Three Major Decision Making Support Systems (Hoivik, 1996) 
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Figure 2. The Acquisition Process Milestone Support Structure (Hoivik, 1996) 
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&. OT&E SUPPORT OF THE ACQUISITION PROCESS 

The acquisition process can take many years from the time a weapon system requirement 
is identified until the weapon system is fielded. As a weapon system progresses through the 
acquisition process, test and evaluation emphasis turns from Developmental Test and Evaluation 
(DT&E) toward Operational Test and Evaluation (OT&E). The fundamental purpose of OT&E in 
the acquisition process 1s to demonstrate the feasibility of the conceptual approach, reduce design 
risk, identify design alternatives, and estimate operational effectiveness and suitability. OT&E 
results weigh heavily on the decisions to continue development, to accept the system, or to 
change organization, doctrine, and concepts that define the weapon system requirements. (DSMC 
TEMG, 1988,p.1-1) Figure 2 depicts the Army T&E events in support of the acquisition 
process. | 

OT&E has an Independent Agency responsibility designed to determine if the weapon 
system meets user requirements. OT&E verifies the effectiveness and suitability of the weapon 
system. The T&E organization, depicted in Figure 3, illustrates its independent nature and 
authority channel. As the weapon system nears the end of the acquisition process, there 1s 
tremendous political and bureaucratic pressure to begin full-scale production. The operational 
testers are often viewed as “show stoppers" because successful completion of OT&E 1s required 
before any weapon system can progress beyond low rate initial production (LRIP). (LeSueur, 
1994, p.12) 
D. THE OT&E ROLE 

OT&E programs are structured to determine the operational effectiveness and suitability 


of a system under realistic conditions (e.g., combat) and to decide if the minimum acceptable 
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operational performance requirements as specified in the Operational Requirements Document 
(ORD) have been satisfied. (DoD 5000.2-R, 1996, p.17) 

Operational performance requirements refer to the weapon system's combat effectiveness 
achieved when operated and maintained by typical users, within the intended doctrine and tactics, 
and as part of standard organizations integrated with other battlefield operating systems. 
Achieved operational effectiveness and suitability are distinct from weapon system capabilities 
(e.g., speed, range, penetration, etc.) determined during DT&E. 

Realism is a critical element of the OT&E process. Realistic combat conditions are those 
that represent the environment, doctrine, level of training, and structure in which the evaluated 


system would normally be expected to operate within a combat environment. 
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E. THE OT&E ORGANIZATION 

Congress passed legislation creating the Office of Operational Test and Evaluation in 
1983 with the mission to independently evaluate the Services' weapon systems tests and assess 
test results. As part of that legislation, the Director of Operational Test and Evaluation 
(DOT&E) reports directly to the Secretary of Defense and is given broad authority to suspend 
major weapons programs that perform poorly in operational tests. In addition to reporting to the 
Secretary of Defense, the DOT&E reports directly to both the House and the Senate Armed 
Services Committees, at least annually, regarding OT&E of weapon systems being developed. 
The goal of the legislation is to save time and money by exposing problems before expensive 
weapons are purchased and fielded. The T&E organization is depicted in Figure 3. 

The Operational Test and Evaluation Command (OPTEC) is the Army's OT&E 
organization and was established 15 November 1990 by Secretary of the Army General Order 
Number 6. The organization, depicted in Figure 4, consists of the OPTEC Headquarters and 
Support Agencies, the Operational Evaluation Command (OEC) and the Test and 
Experimentation Command (TEXCOM). OPTEC's mission is to conduct and monitor user test 
and evaluation (except medical) for the Army. OPTEC's test and evaluation include Initial 
Operational Test and Evaluation and Follow-on Operational Test and Evaluation (IOTE and 
FOTE) in support of the acquisition process, force development testing and experimentation 
(FDTE), concept evaluation program (CEP) trials, early user test and evaluation (EUTE), and the 
Army part of joint test and evaluation (JT&E). 

OPTEC is a field operating agency of the Office of the Chief of Staff of the Army. In 


keeping with the Defense Directives, OPTEC reports the results of Army OT&E through the 
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Vice Chief of Staff of the Army directly to the Army Chief of Staff. The main part of OPTEC's 
mission 1s the planning, conducting, and reporting of Army OT&E required by law since 1972. 
Additionally, OPTEC conducts tests for Training and Doctrine Command (TRADOC) in 
support of its mission to develop combat doctrine, organizations, and material requirements. 
i. OT&E IN EACH PHASE OF THE ACQUISITION PROCESS 

During the Concept Exploration Phase (Phase Zero) prior to Milestone One, laboratory 
- T&E is used to validate, demonstrate and assess the capabilities of key subsystems and 


components. Studies, analysis, simulation, and test data are used to explore and evaluate 
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Figure 4. OPTEC Organizational Structure (OPTEC Homepage, 1996) 
alternative concept designs proposed to satisfy user-generated requirements. OT&E conducted 
curing this phase, calied early operational assessment, investigates deficiencies identified during 


the initial mission area analysis. OPTEC monitors concept exploration for future T&E planning 


iS 


and provides effectiveness and suitability information needed by the program manager. 
Operational assessments that address the operational impact of a weapon system’s technical 
approach are conducted. 

During the Program Definition and Risk Reduction (PDRR) Phase (Phase One), 
operational effectiveness and suitability assessments are conducted. Information on tactics, 
doctrine, personnel requirements, and organizational impacts of the weapon system are gathered. 
OT&E assessments are used to support the Milestone II decisions for developing promising 
alternatives for entering Phase II, Engineering and Manufacturing Development (EMD). 

The objective of the Engineering and Manufacturing Development Phase (Phase Two) is 
to design, fabricate, and test systems that closely approximate the final product. Prior to the 
Milestone III decision, a dedicated IOTE is conducted on the weapon system. This formally 
certifies the weapon system for “final OT&E.” OT&E has the greatest impact on major 
programs during this phase because the decision to proceed beyond LRIP is contingent upon 
successfully completing the IOTE. A formal final Operational Evaluation (OPEVAL) is 
required prior toa Milestone III decision. The specific role of OT&E during Phase Two is to: 

1. Estimate the operational effectiveness and suitability of the system. 

2. Identify operational deficiencies. 

3: Recommend and evaluate changes in production configurations. 

4. Provide information for developing and refining logistics support requirements. 


5. Estimate the survivability of the system in the operational environment. 


Also, during post-production OT&E includes FOTE programs designed to verify the 
operational effectiveness and suitability of the production system and to determine if deficiencies 
identified during IOTE have been corrected. FOTEs also refine doctrine, tactics, techniques, and 
training programs for the life of the system. (DSMC TEMG, 1988) 

G. THE CRITICAL ELEMENTS OF OT&E 

The OT&E process is guided by operationally-related test and evaluation elements. These 
elements are unique to the OT&E process and drive the design of all OT&E programs. Because 
OT&E is designed to test weapon systems under "combat" conditions, these key elements are 
critical to effective testing. (OPTEC 73-21-1, 193) 

1. Realism 

Realistic environments are essential to achieving the goals of OT&E. A realistic 
environment is one that represents the conditions, doctrine, level of training, and structure in 
which the evaluated system would normally be expected to operate in a combat environment. 
These environments are distinctly different from those encountered in DT&E where the 
environment is carefully controlled. 

Realism in the OT&E test process is affected by the resources available to replicate the 
representative threat and friendly force array, create the desired combat battlefield environment, 
and evaluate the system over time. Other factors affecting realism include the degree to which 
test participants represent typical operational personnel, test personnel familiarity with test ranges 
and maneuver areas, and limitations created by safety or environmental concerns. 

The conditions present during OT&E must reach the limits of the evaluated system 


within the doctrinal envelope of operation. Stimulators, external data provided to test the system, 
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are used to stress the limits of the evaluated system. An example of such stimulators would be 
the external data used to stress communications and software-intensive components of an 
evaluated system to levels expected in a sustained combat environment. Effective OT&E ofa 
system also includes testing its interoperability characteristics. This includes creating a realistic 
structure of associated battlefield operating systems to evaluate interoperability. 

2 User Orientation 

An integral member of the OT&E process is the "user." Representative personnel having 
the correct organizational grade and Military Occupational Skill (MOS), level of training, 
experience, and aptitude are required for an unbiased operational evaluation of the system. 
"Users" include personnel that operate, maintain, support, or provide command and control 
functions affecting the evaluated system. The term “user" specifically excludes contractor and 
other personnel who would not normally be involved in the operation, control, or maintenance of 
the evaluated system. To avoid bias, it is important that the personnel selected to represent the 
user are indeed representative. 

3. Representative Systems 

The evaluated system equipment must be sufficiently mature to be considered 
functionally representative of the versions eventually fielded. Software must be complete and as 
near to the fielded version as possible when OT&E is conducted. All associated equipment, 
publications, training programs, and Test Measurement Diagnostic Equipment (TMDE) should 
be complete before the evaluation. The evaluated system is usually part of a larger table of 
organization and equipment and interacts or 1s supported by other standard systems. These 


systems must also be representative of those found in an operational environment. 


4. Sufficiency 

Sufficiency in the OT&E process ensures that the test plan addresses all of the issues 
specified in the Operational Requirements Document (ORD). The data collection plan 
establishes the means for collecting sufficient data and the evaluation of the collected data. The 
structure and use of OT&E methods for acquiring usable information are critical for a complete 
and unbiased evaluation. 

The test plan is the key document in determining what data and under what circumstances 
data are collected. The test plan is driven by the user provided issues specified in the ORD. 
These user issues, termed Critical Operational Issues (COI), drive the test plan to ensure all 
requirements for evaluation are covered in testing. Sufficient, unbiased iterations of test events 
must be accumulated to facilitate the evaluation techniques prescribed in the evaluation plan. 

H. OT&E AND M&S INTEGRATION 

Modeling and Simulation (M&S) in support of OT&E is an emerging application field. 
M&sS supports the OT&E process of weapon systems development as they proceed through the 
acquisition life cycle. Figure 5 depicts M&S application within the acquisition life cycle. 
Currently, the Army uses a wide range of M&S methods in the OT&E process. These include 
engineering models, battlefield environment models, and combat models. M&S can assist 
program managers by reducing OT&E testing costs. Areas for particular savings include: test 
scenario development, operational analysis of weapon systems, sequencing of test events, 
refinement of testing objectives, and development of measures of effectiveness. Additionally, 
M&S can include identifying parameters and drivers for field tests. M&S can assist in 


determining high risk areas, predicting test results, and assisting in testing due to safety and 
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environmental constraints. 

The primary goal of OT&E is operational evaluation. Many OT&E methods, including 
live and simulated testing, are used to reach that goal. M&S can replicate each of the four 
critical elements of OT&E, supporting complete operational evaluation. Because live testing 
carries a significant cost burden, many opportunities are lost to gather valuable data of some 
weapon systems over the life cycle. In many weapon systems, repetitive testing is critical in 
gaining the information needed to completely evaluate the system. M&S provides the ability to 
conducted repetitive testing over time at reduced cost. If OT&E were limited to live testing, it 
would lose much of its opportunity to contribute to decisions on whether to produce a system 
based on extended use. Life cycle costing 1s critical and without including simulation in OT&E, 


knowledge of the life cycle cost would be reduced. (Mann, 1983, p. 6) 
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Figure 5. M&S Integration in the Acquisition Life Cycle (OPTEC Homepage, 1995) 
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OPTEC recognizes the use of M&S to achieve adequate realism, support economical test 
execution, and provide complete and adequate evaluations. However, to ensure that OPTEC 
maintains its objective as an independent evaluator, every M&S application used in the OT&E 
process must be subjected to Verification, Validation, and Accreditation (VV&A). VV&A 
ensues that the simulation used fulfills the four critical elements of OT&E and clearly represents 
a realistic operational evaluation of the weapon system. M&S applications must be accredited 
for each specific application. Because of the VV&A unique requirement, there is no listing or 
catalog of standard M&S applications available for use in the OT&E process. The VV&A 
conducted for one operational test does not apply to another operational test. Integration of M&S 
in OT&E requires advanced detailed planning that allows coordination time for VV&A. 
(OPTEC 73-21-1,1993) 

I. SUMMARY 

The U.S. and the Army are facing a dynamic international environment. The acquisition 
process is the primary means through which the Army procures new weapon systems and 
technology to maintain the “technological advantage” in this dynamic environment. The Army 
acquisition process is designed to field weapon systems that will provide the Army with a 
“technology advantage” in a dynamic warfare environment. Within the acquisition process, 
OT&E assesses the effectiveness and suitability of systems, concepts, doctrines, and tactics in 
realistic combat environments. The results of OT&E assessments have significant impact on the 
acquisition of major weapon systems and have visibility at the Congressional level and the top 
echelons of the Army. M&S techniques can be applied in OT&E to support weapon system 


acquisition. Utilizing M&S to assist in OT&E can aid in fighting the current and future cost 
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constraints that negatively impact on the scope of “live” testing. Currently, M&S takes many 
forms that can span the acquisition process to support the PM in controlling cost, schedule, and 


performance. 
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It. ARMY MODELING AND SIMULATION 


A. GENERAL 

The highest echelons of Army leadership recognize the need to use advanced Modeling 
and Simulation (M&S) capabilities for weapon systems acquisition in the post-cold war era. In the 
United States Army Posture Statement FY95, The Honorable Togo D. West, Jr., Secretary of the 
Army stated: "The Army will maintain technological superiority through pursuit of promising 
advanced technologies and concepts, developing new systems when existing systems have reached 
the end of their useful lives or when a new system offers a revolutionary combat capability. We 
will exploit ... srmulation to support acquisition testing, which will reduce acquisition costs." 
(West, 1994, p.87) Currently, the Army is using over 500 models and simulations in the 
acquisition process to support OT&E. (STRICOM Homepage, January 1997) 

This chapter describes M&S concepts and their application in OT&E. M&S concepts will 
be explained in terms of classes, hierarchies, and hybrid applications. These chosen M&S 
applications are not all-inclusive, but are representative of the technology available. The M&S 
concepts and applications discussed are currently available or are being planned for use by the 
U.S. Army. 
B. CLASSIFICATION OF MODELS AND SIMULATION 

In 1992, the Defense Science Board approved the classification of models and simulations 
depicted in Figure 6. This classification is useful in applying M&S to Army OT&E because it 
allows a simple break-down of complex technologies. M&S applications range from simple 


models, using only one class of M&S, to complex models that integrate all three classes 
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of simulation: constructive, virtual, and live. (DSB, 1993) 


Classes of Models and Simulations 
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Figure 6. Classes of Models and Simulations (DSB, 1993) 


I. Constructive Models and Simulations 

The models and simulations contained within the constructive class currently represent 
the predominant form of M&S used in support of the acquisition process. Within the acquisition 
process, the uses of constructive M&S include design and engineering trade-offs, cost, 
supportability, operational and technical requirements definition, and operational effectiveness 
assessments. Constructive M&S consists of computer models, computer war games, and 
computerized analytical tools which are used by decision makers to evaiuate technica! 


specifications and to identify weapon system shortfalls. Constructive models are used to 
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represent a system and its employment at different levels of detail. At the lowest levels, 
constructive M&sS are used in engineering design testing, costing, and system performance 
calculations. At the highest level, constructive M&S provides information to identify mission 
short falls and to support operational effectiveness testing. (Piplani, 1994, p. 4-2) 

Constructive M&S operates either with or without human interaction. Without human 
interaction, they may be run in multiple iterations to provide higher statistical confidence in 
testing. With human interaction, they validate weapon system operational tests within 
specification and design requirements. 

ja Virtual Simulation 

Virtual simulation brings the weapon system and its operator together in a synthetic or 
simulated environment. Synonyms are often used in describing virtual simulation, which include 
the terms human-in-the-loop, man-in-the-loop, warfighter-in-the-loop, or person-in-the-loop. 
Virtual simulations run in real time, enhancing the data gained from each test iteration. The 
comparatively lower cost of virtual simulation makes it an attractive testing method for the 
program manager. As an example, a virtual weapon system simulator may provide a virtual, but 
nearly-real, crew compartment with the necessary equipment, controls and display panels. A 
computer-generated synthetic environment is then displayed on a screen in front of the crew and 
on the crew compartment instrumentation. Motion of the platform is created by computer 
simulation to replicate the weapon system dynamics. Sounds of the system and equipment can 
also be duplicated. The operators are thereby immersed in an environment created by simulation 
that to the user (or tester) looks, feels, and behaves like the real weapon system. When using 


this type of M&S in testing, the tester operates the equipment, receives commands, and controls 
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the weapons just as in actual combat. (Piplani, 1994, p. 4-3) 

Virtual simulations provide better understanding of human reactions, decision processes, 
and man-machine interfaces. Linked to other simulators, the interaction of multiple weapon 
systems can be examined, leading to changes in tactics or refining weapon system designs. These 
simulations also provide powerful tools for testing actual system hardware and software within 
realistic environments for early OT&E. Additionally, virtual simulation can provide a platform 
for crew training prior to OT&E. 

a. Virtual Prototypes 
A more advanced type of virtual simulation is on our doorstep-- virtual 
prototyping. In this realm, a three-dimensional electronic virtual mockup of a weapon system 
allows an individual to interface with a realistic computer-simulated synthetic environment. The 
representation is solely a computer simulation rather than actual hardware and may be applied in 
early OT&E to evaluate concepts and human-machine interfaces. Additionally, it allows 
designers, logistics engineers, and manufacturing engineers to interface with the weapon system. 
Such an approach supports Integrated Product and Process Development (IPPD) and concurrent 
engineering, by providing a common platform from which all functional disciplines can work. 
The concept of the designer, operator, maintainer and manufacturer all interacting with the same 
realistic three-dimensional representation of the weapon system will become more prevalent in 
future eT (Piplani, 1994, p. 4-4) 
Di Live Exercises 
Live exercises consist of testers and soldiers using equipment under environmental, 


operational, and force structure conditions that replicate actual combat. Live exercises provide a 


testing ground to provide “real” data on actual hardware and software operating in a near-combat 
environment. The introduction of multiple platforms allows for evaluation of interoperability. 
However, assembling the personnel and equipment to conduct a live exercise is a resource 
intensive enterprise requiring time, funds, and people. Data obtained during live exercises can be 
used to validate earlier models and simulations used in testing. This form of testing reinforces 
virtual simulation, ensuring the weapon system supports the mission need. Constructive and 
virtual simulations conducted prior to live exercises can be used to plan the tests, identify critical 
issues, and train the participants. Constructive and virtual simulations may also be used to 
analyze results after the live exercises or to augment tests that may not be feasible to conduct in 
live exercises due to safety or environmental reasons. (Piplani, 1994, p. 4-5) 

Before proceeding further, the reader is cautioned that not every application of M&S fits 
neatly into one of the above classes. The lines among the various classes of M&S are becoming 
blurred as technology advances, allowing interoperability among the classes of M&S. Hybrid 
types of models and simulations are bridging the gaps among two or more classes in support of 
OPE. 

C. HIERARCHY OF MODELS AND SIMULATIONS 

M&sS supports the OT&E process across a wide span of activities ranging from design to 
operational effectiveness assessments. This assortment of tasks requires a suite of models and 
simulations with differing levels of detail, focused at particular applications. The hierarchy of 
M&S depicted in Figure 7 defines the differing levels of detail and their application in OT&E. 


Alongside the hierarchy, a force level and system work breakdown structure (WBS) are shown to 


indicate the system level that corresponds with the level of testing performed. (Piplani, 1994, p. 


4-5) 
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Figure 7. Hierarchy of Models and Simulations (Piplani, 1994, p. 4-6) 


1. Engineering Level Models and Simulations 

Engineering level models and simulations provide the basis for testing design trade-offs 
at the component, subsystem, and system levels. Engineering level M&S support development 
of technical design specifications and OT&E requirements. At the engineering level there are 
literally thousands of models and simulations including: (Piplani, 1994, p. 4-7) 

* Basic models testing phenomenology such as eerodynaimics, tluid flow, 


hydrodynamics, heat transfer, acoustics, fatigue. 
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* Physics-based models of components; subsystem; and systems for design, 

performance, costing, scheduling and supportability. 

Examples of engineering level M&S are cost, support, and manufacturing models. Cost 
models generate development, production, operations, and support costs. Support models 
generate reliability, availability, and maintainability data. Manufacturing models generate 
information - producibility of a particular design, as well as simulation of maintenance work 
flow and requirements. 

Engineering level M&S evaluate performance capabilities, often termed Measures of 
Performance (MOP). Examples of these measures for a missile system may include radar 
acquisition range, miss distance, range, payload or speed. Such performance parameters might be 
used in the system, development, and testing specifications. The representations of a weapon 
system in a higher level M&S have their basis in engineering level models. It is in higher level 
models and simulations that the actual impacts of weapon system performance on combat 
effectiveness, as part of OT&E, are evaluated. 

Pa Engagement Level Models and Simulations 

Engagement level models and simulations represent the system in a combat scenario, 
such as a Or many-on-many. This level of simulation evaluates the effectiveness of an 
individual weapon system against a specific target or enemy threat system. These models provide 
survivability, vulnerability, and lethality results as Measures of Effectiveness (MOEs) for the 


system. 
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The outputs from the use of engagement level M&S indicate the effectiveness of some 
weapon systems in an engagement scenario. Examples for a missile system may include 
probability of kills and/or probability of hits. The uses of engagement level M&S assist the 
program mangers in identifying system effectiveness and performance requirements from the 
Mission Need Statement (MNS) and Operational Requirements Document (ORD). Additionally, 
engineering level M&S is used to conduct system level performance tradeoffs and to evaluate 
weapon ee tactics. 

D. HYBRID MODELS AND SIMULATIONS 

Up to this point, the M&S classes (constructive, virtual and live) and a hierarchy of 
models and simulations have been illustrated. In many applications, combinations among two or 
more classes of M&S are used, resulting in hybrid models and simulations. A hybrid application 
of M&S might employ constructive models to represent a threat in conjunction with actual (live) 
system hardware and software. Examples of such hybrid applications include physical attributes 
simulation, stimulators, and hardware/software in-the-loop (HW/SWIL) simulations, and 
advanced distributed simulations (ADS). 

1. Physical Attributes Simulation 

Physical simulations refer to a physical representation of the actual operating 
environment ae temperature, humidity, shock, vibration). Examples of physical simulation 
include: 

* Munitions shock and vibration testing in a simulated environment. 

* Survivability/ vulnerability evaluations using prototype structures and simulated 


environmental conditions, such as high speed airflow, temperature, and altitude. 
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* A firing impulse simulator (a hydraulically-operated ram) used to provide an impulse to 
the gun barrel to physically replicate the shock of a round being fired for durability and 
shock testing of the artillery piece. 

* A test facility that simulates dynamic loads and motion for evaluation of tracked and 

wheeled vehicle suspension systems. 

These are just some representative samples of physical simulations. Physical simulation 
can be used throughout the OT&E process at component, subsystem and system level for 
evaluating Critical Operational Issues (COI). The data collected from these simulations is useful 
for validating earlier models and simulations. (Piplani, 1994, p. 4-10) 

2. Stimulation 

Stimulations are “non-live” inputs given to weapon systems to test the weapon system, 
just as if the “real” input were present. Stimulations are used because in many instances the 
actual signals representing the “live” environment are not available. These include, for example, a 
radar’s return from a target, a signal from another weapon system, or background noise from the 
environment in which the weapon system must operate. Such stimulations may come from other 
computer models, virtual simulations or from live instrumented tests. 

3. Hardware/Software-In-The-Loop 

The hardware/software-in-the-loop (HW/SWIL) simulations typically span multiple 
classes of simulations. Figure 8 depicts HW/SWIL integration. The HW/SWIL includes actual 
hardware and software with M&S to demonstrate the capability of a weapon system or 
subsystem. An HW/SWIL simulation has proven to be an important tool in system development, 


test and operational support. (Piplam, 1994, p. 4-11) 
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In support of test programs, the HW/SWIL simulations allow for pretest simulation to 
identify test conditions. As a risk reduction measure, they are used for verification of actual 
hardware and software interaction. These simulations are also used to conduct post test analysis 
and to fill a tet matrix for conditions that are either not testable or for which no test assets are 
available. The HW/SWIL simulations allow early identification and correction of developmental 
problems and allow one to identify and focus live tests toward critical issues. With the 


increasing reliance on software within weapon systems, and regular software 
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Figure 8 Hardware/Software-In-The-Loop Simulation (HW/SWIL) (Piplani, 1994, p4-11) 


changes to enhance or modify performance, HW/SWIL has become a primary tool in testing the 


effects of sofiware chanzes on performance. 
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4. Advanced Distributed Simulation 

a. Distributed Interactive Simulation 

One of the challenges in integrating constructive, virtual, and live simulation 
technologies is the use of dissimilar computer systems and simulators languages. Many models 
and simulations were designed independently and were not designed to communicate with one 
another. The integration of simulation capabilities using a common set of standard terms and 
protocols are needed to allow different computer systems and simulators to communicate. The 
standardization of terms is achieved by Distributed Interactive Simulation (DIS) Protocol Data 
Units (PDUs). (Crouch, 1994, p.56) 

DIS creates a synthetic environment within which humans may interact through 
simulation at multiple, networked sites using compliant architectures, modeling, protocols, 
standards, and databases. DIS is the next generation of distributed simulation evolving from the 
Defense Advanced Research Project Agency (DARPA) research project of the 1980’s known as 
Simulation Network (SIMNET). (IST, 1993, p.4) 

DIS will take advantage of current and future simulations manufactured by 
different organizations. The first step in achieving this interoperability is to develop a single 
communications protocol. There must be a standard set of messages that communicate the states 
of simulated and real entities, their interactions, and among host computers. This information is 
communicated through DIS PDUs. The DIS is in its initial stages with many obstacles yet to 
overcome. (IST, 1993, p.8) 

b. Defense Simulation Internet 


DIS operations are supported by a communication system known as the Defense 
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Simulation Internet (DSI). This communication system was developed, and is currently operated, 
by the DARPA. The DSI consists of commercial telephone circuits over existing networks with 
nodes at user locations and strategically placed switching nodes. A central controlling facility is 
located in Chicago, Illinois. Connectivity is established through military and civilian 
communication satellites to allow worldwide, simultaneous DSI operations. 

Each location connected to the DSI network is called a DSI node. There are 
approsiniately 30 DSI nodes supporting all Services' command posts, Battle Simulation Centers, 
test beds, Battle Labs, research centers, unified commands, and civilian contractors that support 
the military. The DSI is expected to expand over the next year with approximately 25 additional 
sites. Currently, the Army operates six core locations which make up the primary backbone of 
the DSI network. The Army has two TRADOC Battle Lab nodes on the DSI; Fort Knox and Fort 
Rucker, with Fort Rucker designated as a Battle Lab support facility. Eventually the Army 
wishes to operate six to eight Army Battle Lab nodes with additional communications nodes at 
major commands throughout the Army. (STRICOM Homepage, 15 January 1997) 

Simulation tools and methodology integration between the Battle Labs and 
industry offer a new area of innovation that has potential to support the acquisition process. The 
use of concurrent engineering principles reduces development time and speeds the acquisition 
process. ame prototypes will enhance early design and manufacturing tradeoffs, leading to 
effective OT&E. Eventually the manufacturing process, the actual weapn system, and the 
system’s performance may all be modeled, integrated, and tested before the first piece of 
hardware is built. (BLG, 1993, p.6) Synthetic environments will not completely replace 


hardware testing. However, given the increased costs of hardware development and testing, 
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contrasted with the decreased costs and advancing technology simulators, synthetic environments 
become ever more appealing. (BLG, 1993, p.7) 
E. SUMMARY 

The Army is moving to take advantage of advanced M&S opportunities and the potential 
cost savings M&S offers in a fiscally constrained post-cold war era. As a vital part of the 
acquisition process, OT&E can integrate M&S concepts and applications to address recurring 
problems and shortcomings in cost, schedule, and OT&E performance. The classes of M&S can 
be included in OT&E planning to help the program manager in reaching cost, schedule, and 
performance goals. With the high cost of live testing, the use of less resource-intensive forms of 
testing, such as M&S, 1s a smart preparation tool. As an integral part of OT&E planning and 
support, M&S allows a program manager to use costly and valuable assets more efficiently. 

The Army will link M&S technology components together through networks such as the 
Defense Systems Internet (DSI) to create environments such as the Distributed Interactive 
Simulation (DIS). Relying heavily upon simulation and modeling technologies, senior Army 
leaders will use M&S as a tool to help save money by speeding up the introduction of promising 
new weapon systems. The Army's leadership is setting policies that endorse M&S use in today’s 


costly acquisition process. 
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IV. ARMY SIMULATION - POLICIES AND REGULATIONS 


A. GENERAL 

The Operational Test and Evaluation (OT&E) process is guided by numerous policies and 
regulations. The provisions of these mandates and guiding vehicles directly affect how OT&E 
for a weapon system is planned, conducted, and analyzed. The role of M&S in the OT&E 
process 1s directed by the policies and regulations guiding OT&E. Initial efforts in providing 
effective policy and regulatory guidance lay the foundation for refining the Army’s future 
processes for managing its M&S investment. Policy and regulatory guidance define how OT&E 
can encompass M&S to streamline the OT&E process. This chapter describes the general Army 
policy and regulations that affect the use of M&S in OT&E. 
B. ARMY AND M&S POLICIES 

The Army is adopting policies to take advantage of advanced M&S opportunities. Army 
leaders are setting the M&S policy cornerstones as evidenced by Army Chief of Staff, General 
Gordon Sullivan’s statement; 

You need to know that we will use simulation techniques throughout the Army’s 
acquisition process. We will determine needs in large-scale, simulation-supported 
exercises that allow us to consider alternative solutions that meet our needs. We 
will use drawings, diagrams, and 3-dimensional models generated by computers, 
put them in constructive or virtual environments, and test alternatives both 
technically and tactically. The most promising technologies will be tested by real 
soldiers, first in reconfigurable crew stations, then in full scale simulations. Final 
designs, production and assembly steps will also be simulated in virtual factories 
before actual prototypes are made. Then the actual and virtual prototypes will be 
tested simultaneously to discover potential problems before production begins. 


Tactics, techniques, and procedures are also developed along with the system so 
that the system is fully ready for use when produced. (Sullivan, 1993) 
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Lieutenant General William H. Forster, the Military Deputy to the Assistant Secretary of 
the Army (Research, Development and Acquisition), further refined the M&S policy in his 24 
May 1993 Memorandum for the Deputy Commanding General, Army Materiel Command and 
all Program Executive Officers. Under the subject of "Simulation Support to the Army,” LTG 
Forster stated, 

The Army Science Board and the Defense Science Board have recently studied 

the potential improvements to DoD acquisition offered by advanced simulation, 

particularly Distributed Interactive Simulation (DIS). Both concluded that 

simulation can improve acquisition from concept to fielding through such 

innovations as: virtual prototyping; engineering simulation; linking of 

constructive, virtual and/or live simulations; assisting the user in execution of 

experiments in employment tactics; user test design and critical issue 

identification; and improved training prior to fielding. (Forster, 1993, p.1) 

Policies integrating M&S into the acquisition process have been initiated from the top 
levels of the Army. Additionally, organizations have been established and charged with pursuing 
new technologies and investigating concepts using an integrated M&S approach. Concepts for 
integrating M&S into the OT&E portion of the acquisition process are coordinated through three 
primary organizations: The Army Model and Simulation Management Office (AMSMO), 
Operational Test and Evaluation Command (OPTEC), and Army Material Systems Analysis 
Activity (AMSAA). 

The Army Model and Simulation Management Office (AMSMO) is the focal point for 
establishing and maintaining the methodology by which to identify and share information among 
agencies, organizations, and activities in the Army community who use or develop M&S. The 


AMSMO promulgates Army M&S policy, publishes guidance, and administers the Army Model 


Improvement Program (AMIP) and Simulation Technology Program (SIMTECH). Additionally, 
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AMSMO develops and publishes the Army Modeling and Simulation Master Plan, implementing 
procedures such as the verification, validation and accreditation (VV&A) of Army models and 
simulations. (AMSMO Homepage, 1996) 

The Operational Test and Evaluation Command (OPTEC) is the Army’s lead agent for 
operational test and evaluation. OPTEC is charged with integrating M&S throughout the testing 
process. OPTEC’s goal is to apply cost-effective techniques to situations of limited testability. 
Additionally, OPTEC incorporates the notion of Distributed Interactive Simulation (DIS) through 
the use of constructive and virtual simulations in operational testing. (OPTEC Homepage, 1996) 

The Army Material Systems Analysis Activity (AMSSA) maintains configuration control 
on M&S for - level performance. This includes one-on-one system performance, many-on- 
many system combat performance, and larger war gaming simulations for OT&E. AMSSA 
uses M&S to provide allied and threat systems performance data for use in cost and operational 
effectiveness testing in Army studies and in support of the acquisition process. AMSSA assists in 
accreditation and provides certified systems performance data characteristics and data to the 
Army testing community. Additionally, AMSSA provides VV&A support to OPTEC and the 
Army Material Command (AMC). (AMSSA Homepage, 1996) 

c. PRIMARY REGULATIONS GOVERNING M&S 

Dramatic advancement in modeling and simulation have made its use in the OT&E 
process more attractive to many program managers. Operational testers are beginning to use 
M&S to support OT&E. Lawmakers are attempting to exploit the advances in M&S by adjusting 


or creating regulations that specify the use of M&S in OT&E. Figure 9 depicts the primary 
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documents that govern the use of M&S in the acquisition process and OT&E 


provides focus to guide the potential M&S user. 


Policies Documents 


United States Code 
Tide 10 
DoDI $000.2 





M&S lavestment DA PAM 73-1 DA PAM 5-11 
Plan 


Figure 9 Documents and Regulations 


I. Title 10, United States Code 


United States Code, Title 10, Chapter 138 has two sections that pertai 





. Each document 


n to the OT&E 


process. Section 2399, Operational Test and Evaluation of Defense Acquisition Programs, and 


section 2366, Major Systems and Munitions Programs: survivability, lethality, and operational 


testing. 


Section 2399 establishes the conditions necessary for a major program (acquisition 


category I and II) to proceed bevond low rate initial production (LRIP). This 


a major defense acquisition program may not proceed beyond LRIP until the 
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provision states that 


Initial Operational 


Test and Evaluation (IOTE) for that program is complete. It further directs that the Director of 
Operational Test and Evaluation (DOT&E) shall analyze the results of major systems' OT&E. 
The resulting report shall state the DOT&E's opinion regarding OT&E adequacy and whether the 
results indicate that the evaluated system is effective and suitable for combat. (USC, 1989, 
pp.230-231) . 

Section 2366 directs the Secretary of Defense to ensure that no major system proceeds 
beyond LRIP until the IOTE survivability and lethality testing is complete. This section 
specifically defines the conditions required for survivability and lethality testing. These 
conditions include the requirement for live fire testing on representative systems and, therefore, 
preclude any major system from being acquired solely on the basis of OT&E conducted using 
only M&S. (USC, 1989, pp.581-582) 

2: Department of Defense Regulation (DoD) 5000.2-R 

This regulation references and reinforces the provisions of Title 10, Unites States Code. 
It directs that OT&E shall be designed to support the decision to proceed beyond LRIP and 
establishes four objectives for OT&E testing: 

1. Provide essential information for assessment of acquisition risk and for 

decision making. 

2. Verify attainment of technical performance specifications and objectives. 

3. Verify that systems are operationally effective and suitable for intended use. 

4. Provide essential information in support of decision making. 

With regard to the use of M&S in the OT&E process, DoD 5000.2-R interprets Title 10 in 


a strict manner. The regulation states that OT&E does not include an operational assessment 


based exclusively on computer modeling and simulation. (DoD 5000.2-R, 1996, p. 15) 
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Sy . DoD M&S Master Plan 

The DoD M&S Master Plan is the Department's first effort in directing, organizing, and 
concentrating M&S capabilities and efforts. It provides a means to achieve a common technical 
and policy consensus. Because of the immense breadth and scope of DoD M&S uses, combined 
with the relative immaturity of many segments of the larger DoD M&S community, a 
comprehensive and complete master plan is difficult to obtain. This plan is intended to be 
dynamic and flexible, a living document that will evolve as M&S technology matures and 
consensus develops on policy and programmatic issues. While many policy and technical issues 
are not identified or resolved, it does, however, provide the management framework and policy 
direction for further implementation of M&S in the acquisition process. Over time, with the 
active participation and support of the DoD M&S community, the DoD M&S Master Plan will 
mature to address the full range of issues confronting M&S applications within DoD. 

4, Army M&S Master Plan 

The Army M&S Master Plan provides information on the modeling and simulation 
vision, standards development process, and objectives for the Department of the Army. It 
represents a statement of the future use of M&S within the Army. The Army M&S Master Plan 
adopts the same M&S objectives as the DoD M&S Master Plan. Additionally, the Army M&S 
Master Plan provides guidance to program managers and operational testers on how to develop 
plans, programs, and projects to incorporate the objectives of the Army in relation to models and 


simulations. 


ra Army Regulation 5-11, Army Model and Simulation Management 
‘Program 


Army Regulation 5-11 (AR 5-11) describes what types of M&S fall under the purview of 

this military regulation, as well as those types of M&S that are excluded. Specifically, M&S 
used for education and training, analysis, test and evaluation (T&E), research and development 
(R&D), or production and logistics are to be responsive to the provisions of AR 5-11. Also 
included are models and simulations that produce input for other models, whose results are then 
used by Army decision makers. M&S developed and/or used by contractors or federally funded 
research and development centers (FFRDC) in support of Army activities must also comply with 
the provisions of AR 5-11. Finally, simulators, semi-automated forces (SAFOR), and models and 
simulations that operate in a Distributed Interactive Simulation (DIS) environments are likewise 
included in this grouping. Models and simulations not falling under the purview of AR 5-11 are 
system training devices, those embedded in weapons systems, machines or tools, and 
instrumentation that does not communicate outside the primary weapon system or device. 

a. DA PAM 5-11 

This new Department of the Army Pamphlet (DA PAM) sets procedures for 
policy contained in AR 5-11, the Army Model and Simulation Management Program. The 
objective of this pamphlet is to assist the M&S developer, proponent, and application sponsor in 
conforming to the verification, validation, and accreditation (VV &A) provisions prescribed in 
AR 5-11. This pamphlet also provides guidance for the development, execution, and reporting of 
all VV&A activities. This pamphlet does not cover data certification procedures, but it does 


address data certification in reference to proper M&S use. DA PAM 5-11 outlines all of the 


documentation, configuration management, certification, and verification and validation 
requirements of data in M&S. 

6. Army Regulation 73-1, Test and Evaluation Policy 

Army Regulation 73-1 (AR 73-1) directs the implementation of DoD Directive 5000.1 
and DoD Regulation 5000.2-R. This regulation describes the type of OT&E application to be 
provided in support of each phase and milestone. It links T&E to the acquisition process and 
directs that T&E be designed to support the acquisition phases and milestones. Within this 
regulation, OPTEC is charged with the overall management of Army OT&E, including the 
integration of M&S where possible to reduce cost, save resources, and verify available data. 

a. DA Pamphlet 73-1 
Department of the Army Pamphlet 73-1 (DA Pam 73-1) suggests the uses of 

models and simulations in test and evaluation (T&E). M&S will be considered to support the 
developmental and operational T&E of software and weapon systems as they proceed through the 
acquisition life cycle. The use of M&S will include, but not be limited to, the identification of 
test parameters and drivers for field tests; determination of high risk areas; prediction of test 
results; assisting in the allocation of scarce test resources; and the assessment of system 
capabilities in situations that cannot be tested due to safety, cost, or other constraints. 
Simulators, emulators, drivers, and stimulators which are used to fully exercise weapon systems 
under operational test conditions are also included. DA PAM 73-1 stipulates that M&S used for 
T&E must be accredited and validated prior to its use for extrapolation or predicting system 
performance (including software, hardware, man-in-loop). The degree to which M&S will 


augment OT&E to assist in weapon system evaluations and assessments will be documented in 
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the Test and Evaluation Master Plan (TEMP). 
| The TEMP is the principal document for test scheduling and planning for the 

program manager and the T&E community. TEMP planning aids decision makers by verifying 
that the weapons systems have attained their technical performance specifications and objectives. 
Specifically, the TEMP ensures that testing will meet operational effectiveness and operational 
suitability requirements as designed by the user. Additionally, the TEMP addresses whether 
existing M&S will be used or whether new M&S will be developed to support OT&E. 
D. REGULATION OVERVIEW 

Army M&s regulations provide the flexibility to allow each acquisition program to tailor 
a T&E strategy applying M&S to achieve maximum program performance. Operational test 
strategies must be generated concurrently with the acquisition strategy to assure that M&S is an 
integral part of the acquisition program. Modeling and simulation will be considered to support 
the technical and operational T&E of all systems (hardware and software) as they proceed 
through the acquisition life cycle. Models and simulations used for OT&E must be accredited 
and validated prior to their use for extrapolation or predicting system performance. The use of 
M&S, whether existing or new, will be documented in the Test and Evaluation Master Plan 
(TEMP) 
E. SUMMARY 

The continuing revolution in advanced technologies has widened the opportunity for 
using M&S in OT&E. This technical revolution has increased the number of users of Army 
M&sS to the point that M&S touches almost all aspects of the acquisition process. To support 


the growing use of advanced technology, policy and regulations have been established to provide 
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a common reference and standard to enhance and focus concurrent use of M&S applications. 
Current policy and regulations are designed to assist the program manager in using M&S 
applications. The development of clear policies and regulations have led to a better 
understanding of the Army M&S management and an increased visibility of M&S activities. As 
advancements in M&S technologies continue, the Army will refine current regulations and 
spawn new organizations that will influence policy. These new organizations will update polices 
and drive lawmakers to revise current regulations, encouraging the use of M&S in OT&E. 
Investments in M&S provide a critical underpinning for a variety of Army missions, from 
designing a future force, building systems and doctrine for that force, and finally to testing 
weapon systems for that force. The current downsizing of the Army creates pressure to reduce 
costs at every level, including within the OT&E process. As the Army continues to increase the 
use of M&S in OT&E, it will see better resource management, cost savings, and enhanced 
verification of testing data. The next chapter illustrates, through case analysis, the results of the 


application of M&S in OT&E. 
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V. CASE ANALYSIS 


A. GENERAL 


The OT&E community has a most difficult role in defining when, where and 

how it can participate outside the live environment. Operational testers find 

themselves caught between United States Code, Title 10 mandates and the spiral 

of diminishing defense dollars. This is a most dangerous position. (Crouch, 1994, 

p.86) 

The "most dangerous position" refers to a situation where the OT&E independent 
evaluator (OPTEC) does not integrate M&S into the testing process but continues to use 
expensive live testing, while program managers (PMs), materiel, and combat developers push to 
incorporate cost effective and realistic M&S applications for OT&E certification. Current 
technological advancements in M&S provide the required simulation fidelity, making M&S a 
viable alternative for some early and follow-on portions of OT&E. (Crouch, 1994, p.87) 

This chapter examines the application of modeling and simulation in OT&E. 
Specifically, three weapon systems will be used to illustrate the viable application of modeling 
and simulation in OT&E. The three primary weapon systems are the M1A2 Abrams Main Battle 
Tank, the Javelin Antitank Missile system, and the AH-64D Longbow Apache Helicopter. 

B. METHODOLOGY 

This analysis will focus on four major OT&E issues: test design validation, resource 
constraints, safety and environmental concerns, and data reliability and validity. Each of the four 
areas is critical to successful OT&E. These major OT&E areas of concern were established by 


conducting interviews with experienced OT&E professionals assigned to the TEXCOM, 


STRICOM, and Program Management Offices. Additional interviews were also conducted with 
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personnel from OPTEC and TRAC Monterey. Interviews were critical in identifying the level of 
M&sS integration for the OT&E programs of each weapon systems examined. The areas of 
concern were Selected when at least two of the three OT&E programs experienced similar 
problems and issues. 
1. Overview of Issues 
The four OT&E areas of concern are test design validation, resource constraints, safety 
and environmental concerns, and data reliability and validity. The general concerns of each area 
and their application in this thesis are described in the following paragraphs. 
a. OT&E Test Design Validation 
Test design issues were categorized as OT&E test design validation concerns when 
shortcomings were associated with the baseline test design. The validation process ensures that 
baseline scenarios and test execution plans satisfy the user and evaluator requirements for data 
collection, while simultaneously meeting the mandates and guidance for the conduct of OT&E. 
A desired method for test design validation is the use of a pilot test. A pilot test is 
a rehearsal of test events using test participants, fully instrumented candidate systems, data 
collection resources, test ranges, and maneuver areas. A pilot test is useful in determining if the 
data collected address the criterion under consideration for specific OT&E events. The pilot test 
advantages include validation of test scenarios, data collection plans, test facilities, and pretest 
training with no test report impact. Disadvantages include time and funding requirements, early 


user familiarization with the candidate system, test facilities, and test trial scenarios. 
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b. Resource Constraints 

Shortcomings associated with time, personnel, equipment, and funding availability 
were categorized as resource constraints. These shortcomings impact the OT&E test design and 
scope in terms of realism, environmental diversity, scale of threat and friendly force structures, 
and number of OT&E event iterations. The degree to which resources are constrained determines 
the operational scope replicated in the live test, including both the type and amount of data 
available for evaluation. 

c: Safety and Environmental Concerns 

OT&E events that were eliminated or unrealistically controlled because of 
potential hazards to personnel or released pollutants into the environment were categorized as 
safety and environmental concerns. As the range, lethality, and destructive ability of associated 
systems all increase with new technologies, safety concerns often limit the use of the actual 
weapon system to very controlled circumstances that are not representative of a combat 
environment. Weapon systems that create radioactivity, release ozone depleting compounds, and 
other environmentally hazardous by-products are limited to controlled scenarios. 

d. Data Reliability and Validity 

Issues associated with test data interpretation or unusual test participant 
qualifications were categorized as data reliability and validity concerns. Live test events contain 
an infinite number of variables, providing an opportunity for assigning cause to undesirable 
outcomes, eliminating outliers, and/or biasing data. Obtaining reliable and valid data for 
evaluation is difficult in an uncontrolled environment. Data reliability and validity are affected 


by both the test design and the operational experience level of the tester. The degree to which 
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test participants represent typical operational users impacts projected force structure and tactical 
unitization of the weapon system for combat operations. 
2. Overview of Primary Weapon Systems 

a. MIA2 Abrams Main Battle Tank 

The M1A2 is the second major block improvement to the Abrams Main Battle 
tank. Enhancements to the M1A1 Abrams include the Inter Vehicular Information System 
(IVIS), Commander’s Independent Thermal Viewer (CITV), Position/Navigation System, an 
Improved Commander’s Weapon Station, and integration of the vehicle electronics system 
through the use of dual redundant data and utility bus architecture. This weapon system is 
designed to be fielded in armored, mechanized infantry, and cavalry units, ranging from platoon 
to division sized elements. (Patten, January 1997) 

b. Javelin Anti-Tank Missile System 

The Javelin (formerly known as the Advanced Antitank Weapon System- 
Medium) is a man-portable, fire and forget, advanced antitank missile system capable of 
defeating all known armor threats. The Javelin is replacing the shoulder-fired Dragon missile 
system and provides extended range and improved attack attitudes. It will replace most Dragon 
systems in combat and combat support units from platoons through corps level. (Knox, January 
1997) 

Cc. AH-64D Longbow Apache Helicopter 

The AH-64 is the Army’s premier attack aircraft, providing direct and indirect fire 
support to both ground and air targets. The Longbow Apache is an upgrade to the AH-64D 


Apache. The enhancements include improved target acquisition and firepower effectiveness, 
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increased survivability and lethality, improved adverse weather fighting capabilities, improved 
armament (including Longbow Hellfire missiles), and automated systems, engines, and fire 
control radar. The Longbow Apache will replace attack helicopters currently fielded in attack 
helicopter units at battalion through corps echelons. (Wolfe, January 1997 ) 
C. ANALYSIS OF M&S APPLICATION IN OT&E 

OT&E problems and issues within each of the identified categories are detailed and the 
capabilities of M&S applications to resolve them are analyzed in the following paragraphs. The 
analysis is organized by issue category. 

1. Test Design Validation 

The operational test design establishes the amount of data and the circumstances in which 
data are collected. Test design lays the foundation for the amount of operational realism 
achieved. Creating realistic combat conditions in OT&E tests can be hampered by many factors: 
accurate interpretation of Test and Evaluation Master Plan (TEMP) critical issues, poor 
understanding of the environments and scenarios, and force structures required to replicate a 
representative combat environment. The M1A2, Javelin, and Longbow Apache test designs had 
shortfalls in test design validation. 

a. MIA2 Main Battle Tank 
The M1A2 live test used an operationally smaller force structure than doctrinally 

employed in combat operations. Operational testing of the M1A2 tank at the brigade or 
divisional levels was not conducted. Operational testing was limited to one battalion-sized task 


force. Battalions deploy, fight, and are supported differently than brigades or divisions. The 


49 


reduced scale significantly increases data assessment extrapolation risks for normal combat 
operations. 

A constructive simulation, such as Janus, appears to have application in addressing 
operational test design shortfalls similar to those experienced by the M1A2. Constructive 
simulations are useful for assessing operational environments and combat scenarios beyond the 
scope of limited live operational testing. Constructive simulations can replicate a wide range of 
force structure, environments, threat forces, and combat intensities. In using constructive 
Simulation to address operational test design issues, live testing can be enhanced across the 
operational spectrum of the battlefield. 

TRAC White Sands Missile Range (WSMR) used a system similar to Janus to 
assist In the M1A2 OTE test design validation. Experiments were designed on a Janus based 
constructive simulation, and experiments were iteratively conducted for predictive analysis. 
Actual test articles were instrumented and data was collected on live test trials. Live test 
iterations were replicated on the simulations and then correlated. The correlated simulations were 
then used in other environments and scenarios within the M1A2 operational spectrum and 
analyzed. The insights gained from these simulations were used to assist in the M1A2 live test 
conducted at TEXCOM Experimentation Center (TEC). Live OT&E test events were selected 
based on the simulated event outcomes that indicated a high probability for obtaining the 
required data to address critical issues. (Payan, 1994) 

b. Javelin Missile 
Battlefield operating systems normally interoperating with the Javelin missile 


systems were not included in the Javelin test plan. Javelin is deployed in conjunction with other 
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anti-armor systems including tanks, artillery, and close air support. An interoperational 
environment was too expensive and impractical to replicate in a live operational test, and was 
omitted from the test design. 

M&s can provide a method for addressing interoperability issues such as those 
experienced by the Javelin during OT&E. Constructive simulations can represent other 
battlefield operating systems, integrating a candidate system’s capabilities so that interoperability 
issues can be assessed. Additionally, virtual simulations can provide a method for assessing 
interoperability issues with personnel operating associated battlefield systems within doctrinal 
guidelines. 

Some M&S was planned into the Javelin OT&E test design. A Model-Test-Model 
(MTM) process conducted on the Javelin program successfully correlated live OT&E test data 
with models generated on Janus. Three scenarios with six missions each were conducted on the 
Javelin model, including offensive and defensive Javelin employments. The resulting data 
provided Javelin employment assessments in an operational environment. OT&E test scenarios 
were changed to incorporate the MTM recommendations. Integrating MTM efforts into future 
OT&E test designs can improve test scenarios and environments, and therefore, enhance 
operational assessments. (IR MTH, 1993) 

Cc. AH-64D Longbow Apache 

During comparative performance assessment between the AH-64D and the AH- 
64A, threat forces were realistically deployed and represented a mid-intensity conflict. 
(Stanfield, March 1997) However, the AH-64D Longbow Apache OT&E test design had threat 


realism shortfalls for high intensity conflicts. After the IOT&E comparative assessement, further 
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test designs did not replicate many of the threat weapons expected in a combat environment. 
Planned threat air defenses were under-represented and lacked realistic electronic jamming 
capabilities. Additionally, threat air assets were under-represented and excluded from many 
planned test scenarios. 

M&s can provide methods for addressing threat realism shortfalls in live OT&E 
test designs. Constructive simulations can replicate the full range of threat capabilities expected 
in acombat environment. Additionally, virtual simulations, incorporating Semi-automated 
Forces (SAFOR), can provide a means to include man-in-the-loop threat forces employed by 
soldiers using doctrinally correct threat tactics. 

M&S applications were used in the Longbow Apache OTE test design process. 
TEXCOM validated the scenarios for the test plan using Janus constructive simulations. In 
addition, the Longbow Hellfire missile was examined in various electronic warfare counter- 
measure situations using a hardware-in-the-loop simulation. These pretest simulations were used 
to establish Hellfire probability of hit (Ph) and probability of kill (Pk) for casualty assessments. 
The success of M&S integration in the Longbow Apache operational test indicates that M&S 
application can improve the test design process. 

d. Test Design Summary 

In many operational tests, the live test represents only a small part of the actual 
combat environment for the weapon system. As a result, the data collected and the test 
evaluations represent only a part of the operational spectrum. Data for evaluation in larger 


operational assessments, beyond the live test design, must be extrapolated to address the 
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weapon system’s combat effectiveness and suitability. As the scope of live testing is reduced, 
increases in data extrapolation reduce evaluation validity. 

M&S integration into the OT&E test design appears to provide an opportunity for 
addressing areas that are beyond the scope of the planned live test. M&S applications, including 
MTM, constructive simulations, and virtual simulations can enhance the OT&E test design 
process. Candidate systems in a constructive or integrated simulation can be portrayed at every 
echelon in the Army structure, under varying physical and threat environments, and 
interoperating with other battlefield and logistics systems. 

2 Resource Constraints 

A declining DoD budget adds pressure on PMs and testers to limit the scope of 
operational testing. Under funding and limited resources contribute to reduced realism in OT&E. 
Resource constraints had a serious negative impact on the MIA2 and Javelin tests. 

a. MIA2 Main Battle Tank and Javelin Missile 

Due to lack of congressional funding and limited test ranges, the M1A2 and the 
Javelin were tested at the small unit level. The horizontal interoperability and vertical impact on 
higher echelon units could not be ascertained by the OT&E tests conducted. Resource constraints 
limited the inclusion of associated battlefield operating systems and support elements that would 
normally be deployed in a combat environment. Interoperability evaluation could not be 
addressed from test data collected. Realism in live testing goes beyond the small unit. For 
example, recovering and evacuating an M1Az2 tank from the battlefield to the corps level general 
support maintenance unit requires significant recovery and transportation assets as well as multi- 


echelon coordination. M1A2 OT&E test resource constraints did not allow this scenario to be 
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replicated beyond the participating battalion’s organic recovery capabilities. The Javelin 
experienced similar problems in resupply of missiles to forward troops during high threat combat 
environments. In the Javelin Live test, the General Support unit was not utilized. 

Interoperability with other systems, beyond the scope of live testing, can be 
partially assessed through M&S. For example, the TRAC White Sands Missile Range (WSMR) 
simulations addressed M1A2 capabilities in various environments and scenarios beyond the 
scope of live testing. Using simulation applications, the M1A2 could be portrayed interoperating 
with close air support, artillery and any other battlefield operating system including critical 
logistics support activities. The capability to fully replicate interoperability, both horizontal and 
vertical, is currently outside the resource-constrained OT&E environment. 

Resource constraints have a considerable impact on the planning of live tests. 
However, M&S has application in addressing the OT&E test shortfall created by resource 
constraints. The marginal cost of M&S generated operational events is extremely low compared 
with those same events conducted in the live environment. An example of the wide disparity 
between live and M&S resource requirements involves M1A1 main battle tank prototypes. In 
1984, evaluations of possible improvements on the M1A1 Abrams tank were carried out by using 
real tanks in a live environment. The effort took 24 months and cost $40 million. A later effort in 
1986 used a modified aircraft dome simulator, took only six months and cost $1 million. In 1992, 
using Distributred Interactive Simulation (DIS), four variations of the M1A1 Abrams were 


operated against potential threats, taking only three months and costing $640, 000. (Berry, 1992) 
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b. Resource Constraint Summary 

Resource limitations impact realism and reduce the scope of the operational 
testing. Reduced realism in operational testing results in less quantitative and more qualitative 
evaluations of the candidate weapon system. Reduced scope in operational testing increases 
data extrapolation for larger force-on-force evaluations. 

Virtual and constructive simulations can enhance OT&E tests that have been 
negatively affected by resource constraints. Virtual simulation can provide methods for 
representing threat and friendly force capabilities that are too expensive or impractical to 
replicate in a live operational test environment. Constructive simulations can provide a means to 
assess portions of the operational spectrum outside the scope of live testing. 

5: Safety and Environmental Concerns 

Safety and environmental impacts of new technology and hazardous complex compounds 
adversely affects the successful creation of a realistic combat environment for operational testing. 
Peacetime requirements often limit the amount and type of testing that can be accomplished, 
unlike the combat environment where safety and environmental side effects are not so important. 
Safety and environmental issues affected the M1A2, Javelin, and Longbow Apache OT&E tests. 

a. Range Limitations 

A common problem in conducting live operational tests are the peqacetime range 
limitations. Range limitations affected the operational employment of the M1A2, Javelin, and 
Longbow Apache systems during OT&E. The ranges used had well marked firing limits and 
oriented the crews toward target positions. Lasers, for all but the Javelin Missile, were restricted 


to controlled maneuver and air spaces. Controlled maneuver and air spaces did not provide 
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representative maneuver space or engagement ranges. Environmental concerns eliminated 
operational firing of depleted uranium M1A2 main gun rounds due to radiation hazards. 

Both constructive and virtual simulation applications provide environments that 
are not affected by these types of safety and environmental range limitations. As a result, tests 
generated through M&S can avoid such biased factors. 

b. Crew Safety Requirements 

Both PM and TEC test officers strictly followed peacetime crew rest requirements 
for helicopter pilots, tank crews, and ground soldiers. These crew rest requirements are not 
representative of typical combat crew duty cycles, which are not constrained to peacetime 
training restrictions. Additionally, during OT&E the weapons systems were not tested to the 
normal operational readiness factors associated with combat operations. 

In a protected simulation environment, where there are no actual threats to the 
safety of soldiers and where safety considerations are nearly nonexistent, M&S applications can 
be used to assess the impact of extended combat operations on aircraft and ground crews. For 
example, relatively close air-to-air combat involving more than the safety constrained two-on- 
two force ratios are not probable in an operational environment. Virtual or constructive 
simulations provide a means to address issues involving more than two-on-two force ratios. 
Crew safety concerns make simulation a viable and practical method for obtaining this type of 
operational data. 

Cc. Safety and Environmental Concerns Summary 

Safety and environmental concerns negatively impact on the ability to assess the 


operational effectiveness and suitability of candidate weapon systems. Integrating M&S into the 
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OT&E process provides a means for assessing issues in an environment that is not constrained by 
safety and environmental concerns. Virtual and constructive simulation applications can be 
integrated into operational tests that cannot be safely or environmentally conducted in a live 
operational environment. Additionally, when test events are limited in number due to safety or 
environmental concerns, model-test-model (MTM) can provide a method to extend data and 
enhance operational evaluations. 

4. Test Data Reliability and Validity 

The myriad of variables present in a live test environment impact on data reliability and 
validity. One of the most common concerns in obtaining reliable and valid test data is how well 
test personnel represent typical users at the operational level. The M1A2 and Longbow Apache 
experienced data reliability and validity concerns due to crew experience levels that were above 
typical tactical unit proficiency levels. Experience level of test crews centers on two issues: 
1)Pre-training of test personnel for the specific OT&E event, 2) operational expereince of test 
participants. OT&E personnel at TEC indicated that the M1A2 and Longbow Apache OT&E 
had problems associated with atypical test participants. For both weapon systens, OT&E test 
participants were tasked from tactical units for the test duration. 

a. MIA2 Test 
Training and familiarization of the M1A2 test crews were conducted on the same 

ranges and maneuver areas as the OT&E test trials. Decisions, response times, and accuracy of 
weapons firing were considered superior to what could be expected in a combat environment 


because of familiarity with the test ranges. Many of the battle position were pre-designated and 
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maneuver was artificially controlled to meet range requirements. Pre-testing training and range 
control procedures contribute in making M1A2 test participants atypically qualified. 

b. AH-64D Apache 

Because of the complex nature of the AH-64D, the operators were selected from 
the same operational unit, A/2-229th. (Stanfield, March 1997) Using all of the test participants 
from an Single unit could bias test data. For example, of the sixteen pilots for the operational test, 
the majority were senior chief warrant officers (CW3 or CW4) who have worked together over 
long periods of time. Their pilot skills were considered to be atypically high when compared 
with less experienced pilots prevalent in the operational units. Additionally, ranges at Hunter 
Liggett required the use of common battle positions for testing. Again, pre-test training and 
familiarization with range procedures will enhance the crews ability to quickly and effectively 
engage targets from designated battle positions. 

The Distributive Interactive Simulation (DIS) network provides an opportunity to 
address the atypical crew problem. Test participants selected over a wide geographic area can be 
linked together through DIS and participate in OT&E testing without leaving their home station. 
The tendency to select atypical users at home station would be reduced because participants 
remain anonymous and are not representing the unit as individuals. The same techniques could 
be used to train test participants without familiarizing them with the limited test ranges and 
maneuver areas used for live test events. Data collected in this manner would enhance reliability 


and validity, thus improving the evaluation. 
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c. Test Data Reliability and Validity Summary 

Data reliability and validity are affected by the OTE test scope, realism 
achieved, and the degree to which test participants represent typical users. Operational 
assessments of combat effectiveness and suitability are directly affected by data validity and 
reliability. Integrating M&S into the OT&E process through the use of model-test-model (MTM), 
virtual, constructive, and Distributed Interactive Simulations (DIS) enhances data reliability and 
validity, and therefore, weapon system evaluations become more accurate. 

The uniform quality of M&S-generated data appears to enhance the reliability and 
validity of OT&E data. M&S-generated data are less susceptible to bias because the variables 
are controlled by design. Additionally, testing can be replicated for reliability and validity. 
Fully verified, validated and accredited M&S applications specifically designed for data 
collection produce uniform quality data which should enhance the evaluation process. 

D. SUMMARY 

Integration of advanced M&S techniques, technologies, and applications into OT&E can 
provide a means of addressing operational test issues beyond the scope of resource-constrained 
live testing. M&S provides a vehicle for addressing critical issues that are impossible or 
impractical to represent in a live environment. Operational realism and combat stress may be 
enhanced through M&S application. 

Integrating M&S does not replace or eliminate live operational testing, but rather 
augments and focuses live testing. The adage "you dont use M&S to identify what is exactly 


right, you use it to eliminate what is exactly wrong " applies to OT&E test design. Integrating 
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M&S into OT&E design reduces variability by examining critical issues across the operational 


spectrum replicated on a simulation. 
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VI. CONCLUSIONS AND RECOMMENDATIONS 


A. OVERVIEW 

The OT&E process has a major impact on a weapon systems’ critical acquisition 
decisions. The importance of OT&E in the acquisition process is underscored by Congressional 
interest in the OT&E process and it’s results. OT&E assessments are reported directly to 
Congress from independent OT&E sources. From early operational assessments through the 
formal operational evaluation, the insights to a weapon systems’ characteristics are based on 
operational tests. The accuracy of OT&E in predicting how weapon systems will operate in a 
combat environment directly affects future combat implementation, and more importantly, the 
survival of the combat troops who depend upon those weapon systems. 

M&sS integration into the OT&E process has traditionally been limited. In the past, M&S 
maturity was limited to physical models and mathematical simulation. Advancements in 
software, hardware, and linked communications networks have resulted in explosive M&S 
growth and maturity. However, the OT&E community is slow to integrate advanced M&S due to 
the desire to protect their independent status and the perceived negative connotation associated 
with anything "simulated" in a live test environment. 

This thesis focused on the potential opportunities to integrate M&S concepts and 
applications into the Army OT&E process. Recurring OT&E concerns were examined and 
possible M&S based enhancements and solutions were identified. Increased M&S integration in 


the weapons testing will enhance the OT&E process. 
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B. GENERAL CONCLUSIONS 

Realistic operational combat effects on personnel and equipment cannot be wholly 
replicated through constructive simulation or virtual simulation in operational testing. Live 
testing in a replicated combat environment provides the most realistic operational test, however, 
it is the most expensive OT&E method. A coordinated approach to operational testing, 
integrating live tests with models and simulation M&S, provides a realistic environment for 
effective OT&E. M&S cannot, and should not be used in place of necessary OT&E live testing. 
However, M&sS applications offer methods of augmenting and enhancing the OT&E process that 
merit consideration. 

The tremendous costs associated with conducting comprehensive live testing will limit its 
use in the budget restrained future. Army budgets are projected to continue the present 
downward trend, which will not improve the prospects for expanding OT&E live testing. The 
requirement to create holistic, representative operational environments conflicts with the realities 
of the reduced current budget and projected defense funding authorizations. M&S applications 
can bridge the funding gap, assisting both the PM and operational testers in providing good 
equipment for the user. 

Interviews and research indicate there are significant problems and concerns associated 
with the Army OT&E process. Some believe that failures during OT&E are management 
failures, rather than learning opportunities. Many focus on the inability of limited OT&E to 
adequately assess the combat effectiveness and suitability of weapon systems, as required by 
law. Whatever the reason for limited test, it can lead to the fielding of weapon systems whose 


operational effectiveness and suitability have not been fully demonstrated. Inadequate 
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operational testing tends to yield more favorable assessments than are likely to be found when 
the systems are employed in combat. 

The integration of M&S can provide some solutions to recurring OT&E problems and 
issues. Integrating M&S into the OT&E process can provide valuable insights into the 
operational environment beyond the scope of a resource constrained OT&E process. However, 
M&sS cannot fully represent the personnel stresses associated with combat environments. 
Additionally, M&S cannot replicate the infinite number of variables present in live 
environments. 

c. SPECIFIC CONCLUSIONS 

1. Test Design 

Interviews with PM and OT&E professionals indicated that during OT&E it is 
impossible or impractical to portray the full range of force structures. Demonstrating a weapon 
system’s integration with friendly forces and interoperation with potential threat capabilities is 
difficult for even the most basic system. This is largely due to the fact that current methods for 
planning OT&E and validating that plan are not efficient or effective. 

Integrating M&S into the OT&E test design process can assist the PM and test planners in 
addressing the problems and issues associated with test design. Constructive simulations can 
provide a means of examining the candidate weapon system's entire operational spectrum. 
Force structures, interoperating systems, and threat capabilities can be represented in the 
constructive environment. Integrating live test events with constructive simulations, such as 
Model-Test-Model applications, can identify scenarios and environments required to address 


critical testing issues. 
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2. Resource Constraints 

Resource constraints limit the scope of live OT&E testing. The operational test often 
shrinks as resources are constrained. Resource constraints limit the degree to which realism is 
replicated in a live test environment. These constraints increase the risk that the 
OT&E will neither accurately nor adequately address the combat environment that 
potential weapon systems might experience. 

Integrating M&S into OT&E testing can reduce the impact created by resource 
constraints. Constructive and virtual simulations can provide methods to address operational 
issues beyond the scope of resource constrained live testing. M&S applications are more cost 
effective than live test events and, therefore, are less susceptible to the negative effects resulting 
from resource constraints. Virtual simulations accommodate actual users in an unconstrained 
environment. This provides critical user responses 1n environments with realistic threat 
capabilities, interoperability with other battlefield operating systems, and force structures 
impossible to replicate in a live environment. 

3. Safety and Environmental Concerns 

In today’s testing environment, safety and environmental issues increasingly limit the 
ability to fully consider critical issues in a live OT&E. New technology advancements including 
lasers, depleted uranium projectiles, and other potentially hazardous systems cannot be addressed 
in an unrestricted, live operational test. Systems that produce environmentally hazardous by- 
products are restricted to very controlled scenarios, not representative of a combat environment. 
Additionally, peacetime safety requirements, significantly more restrictive than those expected 


in a combat environment, negatively impact OT&E assessments. 
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Integrating M&S into the OT&E process can provide solutions for data collection when 
scenarios are restricted by safety or environmental concerns. M&S can provide a means to 
address critical issues that are too hazardous or environmentally polluting to conduct in a live 
test. In extreme situations, M&S may provide the only OT&E data obtainable for OT&E 
evaluation. 

4. Data Reliability and Validity 

Limited operational testing creates data reliability and validity problems and issues. Data 
collected from limited OT&E must be extrapolated to the operational spectrum not addressed 
within the scope of live OT&E. As the OT&E test scope becomes smaller and more constrained, 
data reliability and validity risks increase. The myriad of variables present in a live OT&E 
environment invites questionable interpretation of the results. False causes may be assigned to 
some undesirable events that eliminate, obscure, or reduce the certainty of the failure. 

OT&E test participants are often not representative of the typical operational user. Data 
reliability and validly suffer when "golden crews” (personnel who are atypically high in aptitude 
and attitude) are used to represent the "typical" user. Training test participants on the same 
ranges and maneuver areas used for OT&E test trials creates unrealistic responses. This 
familiarity improves response times, anticipation, and weapon accuracy. 

Integrating M&S into the OT&E process can assist in addressing data reliability and 
validity related problems and issues. M&S applications can provide a means to augment data 
required for quantitative evaluation. Data generated on M&S applications have controlled 
variables, virtually eliminating the opportunity for event interpretation and manipulation. DIS 


can provide an effective means for addressing the "golden crew " problem. Data collection and 
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test participant training via DIS preserve the actions, reactions, and decisions of representative 
operational users. 
D. RECOMMENDATIONS 

The following recommendations, based on weapon system analysis, should enable M&S 
to enhance and improve the Army OT&E process: 

Ie The Army should continue to expand existing M&S efforts in requirements 
generation, data augmentation, and OT&E testing. Cooperative M&S efforts between OPTEC 
and TRADOC will enhance the OT&E process through early and accurate critical issue 
identification and test design validation. As a result, OT&E test data will be improved and 
combat effectiveness and suitability assessments will be more accurate. 

a M&S applications that have the fidelity to integrate candidate system capabilities 
and parameters for OT&E should be developed. The M&S applications developed should 
portray existing threat and friendly force structures from small unit through theater level 
environments. These applications should include constructive and virtual simulations that can 
examine the impacts on threat capability, battlefield interoperability, and logistics supportability 
issues. Accurate simulations provide methods for examining the operational spectrum beyond the 
live test scope. Extrapolation of live test data via M&S can enhance data and evaluation 
accuracy. 

8). Verification, Validation and Accreditation (VV&A) requirements for previously 
validated applications should be limited to the candidate systems’ capabilities and parameters. 
The VV&A process for M&S applications is time consuming and expensive. M&S applications 


with pre-approved VV&A accreditation could enhance M&S integration into the OT&E process. 
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4. Equipment simulators at user locations should be linked through DIS. Virtual 
simulations, linked through the DIS network, would integrate a wide variety of typical users for 
operational assessments. Test participant training at home station through DIS could improve 
OT&E data validity during live test trials. 

E. AREAS FOR FURTHER RESEARCH 
Further research is recommended in the following areas: 

1. Research should be conducted to determine the differences between test participant 
actions, reactions, and decisions in live versus simulated environments. The results of this 
research should help establish the degree to which M&S generated reaction and decision data 
correlate with live operational testing. 

2. The integration of high fidelity M&S applications should be analyzed. Candidate 
system employment via high fidelity simulations could provide effectiveness and suitability 
assessments while varying doctrine and tactics. Additionally, threat forces controlled through 
high fidelity simulations could provide probable threat adjustments to the employment of 
candidate weapon systems. 

3. The compatibility of current and future weapon systems training simulation devices 
with DIS and other M&S architectures should be investigated. Dual use of training simulation 
devices for training and OTE testing via virtual and distributed simulations could enhance the 


M&s integration into the OT&E process. 
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APPENDIX 


ADS- Advanced Distributed Simulations 

AMC- Army Material Command 

AMIP- Army Model Improvement Program 

AMSAA- Army Material Systems Analysis Activity 

AMSMO- Army Modeling and Simulation Management Office 
AR- Army Regulation 


BDS- Battlefield Distributed Simulation 
BDS-D- Battlefield Distributed Simulation-Developmental 


C&C- Command and Control 

C4]- Command, Control, Communications, Computers, and Intelligence 
CEP- Concept Evaluation Program 

CITV- Commander’s Independent Thermal Viewer 

COI- Critical Operational Issue 

CW3- Chief Warrant Officer - 3 

CW4- Chief Warrant Officer - 4 


DA PAM- Department of the Army Pamphlet 
DARPA- Defense Advanced Research Project Agency 
DIS- Distributed Interactive Simulation 

DMSO- Defense Modeling and Simulation Office 
DoD- Department of Defense 

DOT&E- Director of Operational Test and Evaluation 
DSB- Defense Science Board 

DSI- Defense Simulation Internet 

DT&E- Developmental Test and Evaluation 


EMD- Engineering and Manufacturing Development 
EUTE- Early User Test and Evaluation 


FDTE- Force Development Testing and Experimentation 
FFRDC- Federally Funded Research and Development Center 
FOTE- Follow-on Operational Test and Evaluation 
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HW/SWIL- Hardware/Software-in-the-loop 


IOTE- Initial Operational Test and Evaluation 
IPPD- Integrated Product and Process Development 
IVIS- Intervehicular Information System 


JT&E- Joint Test and Evaluation 


LAM- Louisiana Maneuvers 
LRIP- Low Rate Initia] Production 


M&S- Modeling and Simulation 

MAIS- Major Automated Information systems 
MDAPS- Major Defense Acquisition Programs 
MNS- Mission Needs Statement 

MOE- Measure of Effectiveness 

MOO- Measures of Outcome 

MOP- Measure of Performance 

MOS- Military Occupational] Skill 

MTM- Model-Test-Model 


OEC- Operational Evaluation Command 

OPEVAL- Operational Evaluation 

OPTEC- Operational Test and Evaluation Command 
ORD- Operational Requirements Document 

OT&E- Operational Test and Evaluation 


PDRR- Program Definition and Risk Reduction 
PDU- Protocol Data Unit 

Ph- Probability of Hit 

Pk- Probability of Kall 

PM- Program Manager 

PPBS- Planning, Programming, and Budgeting 


R&D- Research and Development 


70 


SAFOR- Semi-automated Forces 

SIMNET- Simulation Network 

SIMTECH- Simulation Technology Program 

STRICOM- Simulation, Training and Instrumentation Command 


T&E- Test and Evaluation 

TEC- TEXCOM Experimentation Center 

TEMP- Test and Evaluation Master Plan 

TEXCOM.- Test and Experimentation Command 

TMDE- Test Measurement Diagnostic Equipment 
TRAC- Training and Doctrine Command Analysis Center 
TRADOC- Training and Doctrine Command 


VV&A- Verification, Validation, and Accreditation 


WBS- Work Breakdown Structure 
WSMR- White Sands Missile Range 
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